{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#all_skip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from fastai.text.all import *\n",
    "from nbdev.showdoc import show_doc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ULMFiT"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Finetune a pretrained Language Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First we get our data and tokenize it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "path = untar_data(URLs.IMDB)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "100000"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "texts = get_files(path, extensions=['.txt'], folders=['unsup', 'train', 'test'])\n",
    "len(texts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we put it in a `Datasets`. For a language model, we don't have targets, so there is only one transform to numericalize the texts. Note that `tokenize_df` returns the count of the words in the corpus to make it easy to create a vocabulary."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def read_file(f): return L(f.read_text().split(' '))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "splits = RandomSplitter(valid_pct=0.1)(texts)\n",
    "tfms = [Tokenizer.from_folder(path), Numericalize()]\n",
    "dsets = Datasets(texts, [tfms], splits=splits, dl_type=LMDataLoader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we use that `Datasets` to create a `DataLoaders`. Here the class of `TfmdDL` we need to use is `LMDataLoader` which will concatenate all the texts in a source (with a shuffle at each epoch for the training set), split it in `bs` chunks then read continuously through it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs,sl=256,80\n",
    "dbunch_lm = dsets.dataloaders(bs=bs, seq_len=sl, val_bs=bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>text_</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>▁xxbos ▁xxmaj ▁this ▁is ▁an ▁xxmaj ▁emperor ' s ▁xxmaj ▁new ▁xxmaj ▁clothes ▁situation . ▁xxmaj ▁someone ▁needs ▁to ▁say ▁\" that ' s ▁not ▁a ▁funny ▁and ▁original , ▁( etc . , ▁etc . ) ▁film ; ▁that ▁is ▁an ▁inferior ▁film . ▁xxmaj ▁don ' t ▁waste ▁your ▁money ▁on ▁it .\" ▁xxmaj ▁the ▁film ▁is ▁trashy , ▁and ▁the ▁people ▁in ▁it ▁are ▁embarrassingly ▁inferior ▁trailer ▁trash . ▁xxmaj ▁they ▁are ▁all - too - realistic</td>\n",
       "      <td>▁xxmaj ▁this ▁is ▁an ▁xxmaj ▁emperor ' s ▁xxmaj ▁new ▁xxmaj ▁clothes ▁situation . ▁xxmaj ▁someone ▁needs ▁to ▁say ▁\" that ' s ▁not ▁a ▁funny ▁and ▁original , ▁( etc . , ▁etc . ) ▁film ; ▁that ▁is ▁an ▁inferior ▁film . ▁xxmaj ▁don ' t ▁waste ▁your ▁money ▁on ▁it .\" ▁xxmaj ▁the ▁film ▁is ▁trashy , ▁and ▁the ▁people ▁in ▁it ▁are ▁embarrassingly ▁inferior ▁trailer ▁trash . ▁xxmaj ▁they ▁are ▁all - too - realistic ally</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>▁xxmaj ▁listener ▁is ▁without ▁doubt ▁one ▁of ▁the ▁dullest ▁films ▁i ▁have ▁ever ▁seen . ▁xxmaj ▁there ▁was ▁nothing ▁happening ▁in ▁this ▁film ▁what ▁so ▁ever ▁- ▁i ▁didn ' t ▁care ▁for ▁any ▁of ▁the ▁characters , ▁didn ' t ▁buy ▁in ▁to ▁the ▁whole ▁mystery ▁type ▁plot , ▁didn ' t ▁care ▁how ▁it ▁ended ▁.... nothing . ▁xxmaj ▁there ▁is ▁no ▁comedy , ▁no ▁action , ▁no ▁thrills , ▁no ▁suspense , ▁nothing . ▁xxmaj ▁the ▁highlights</td>\n",
       "      <td>▁listener ▁is ▁without ▁doubt ▁one ▁of ▁the ▁dullest ▁films ▁i ▁have ▁ever ▁seen . ▁xxmaj ▁there ▁was ▁nothing ▁happening ▁in ▁this ▁film ▁what ▁so ▁ever ▁- ▁i ▁didn ' t ▁care ▁for ▁any ▁of ▁the ▁characters , ▁didn ' t ▁buy ▁in ▁to ▁the ▁whole ▁mystery ▁type ▁plot , ▁didn ' t ▁care ▁how ▁it ▁ended ▁.... nothing . ▁xxmaj ▁there ▁is ▁no ▁comedy , ▁no ▁action , ▁no ▁thrills , ▁no ▁suspense , ▁nothing . ▁xxmaj ▁the ▁highlights ▁include</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>▁mutated ▁humans ▁who ▁have ▁been ▁outcast ▁by ▁society . ▁xxmaj ▁eventually , ▁they ▁receive ▁a ▁special ▁en vo y ▁from ▁xxmaj ▁earth ▁with ▁an ▁unexpected ▁message . ▁xxmaj ▁the ▁basic ▁problem ▁is ▁that ▁this ▁whole ▁movie ▁could ▁have ▁been ▁summarized ▁into ▁a ▁sentence ▁and ▁making ▁a ▁1 ▁hour ▁movie ▁out ▁of ▁it ▁added ▁nothing . ▁xxmaj ▁what ▁you ▁essentially ▁get ▁is ▁some ▁effectively ▁gross - looking ▁characters ▁with ▁dialog ▁that ▁is ▁so ▁boring ▁you ▁want ▁to ▁blow ▁their ▁ship ▁up</td>\n",
       "      <td>▁humans ▁who ▁have ▁been ▁outcast ▁by ▁society . ▁xxmaj ▁eventually , ▁they ▁receive ▁a ▁special ▁en vo y ▁from ▁xxmaj ▁earth ▁with ▁an ▁unexpected ▁message . ▁xxmaj ▁the ▁basic ▁problem ▁is ▁that ▁this ▁whole ▁movie ▁could ▁have ▁been ▁summarized ▁into ▁a ▁sentence ▁and ▁making ▁a ▁1 ▁hour ▁movie ▁out ▁of ▁it ▁added ▁nothing . ▁xxmaj ▁what ▁you ▁essentially ▁get ▁is ▁some ▁effectively ▁gross - looking ▁characters ▁with ▁dialog ▁that ▁is ▁so ▁boring ▁you ▁want ▁to ▁blow ▁their ▁ship ▁up ▁every</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>▁soul ▁of ▁man ▁can ▁be ▁distorted ▁in ▁such ▁a ▁way ▁that , ▁pain ▁and ▁suffering ▁being ▁brought ▁to ▁bear ▁on ▁a ▁fellow ▁human ▁being ▁is ▁in ▁some ▁way ▁satisfying . ▁xxmaj ▁be ▁it ▁mental ▁or ▁physical . ▁i ▁found ▁the ▁film ▁very ▁thought ▁provoking . ▁xxbos ▁i ▁have ▁been ▁hooked ▁on ▁\" gg \" ▁since ▁midway ▁through ▁2001 - 2002 ▁(2 nd ▁season ), ▁when ▁i ▁tuned ▁in ▁to ▁see ▁\" small ville \" ▁10 ▁minutes ▁early . ▁xxmaj ▁thanks</td>\n",
       "      <td>▁of ▁man ▁can ▁be ▁distorted ▁in ▁such ▁a ▁way ▁that , ▁pain ▁and ▁suffering ▁being ▁brought ▁to ▁bear ▁on ▁a ▁fellow ▁human ▁being ▁is ▁in ▁some ▁way ▁satisfying . ▁xxmaj ▁be ▁it ▁mental ▁or ▁physical . ▁i ▁found ▁the ▁film ▁very ▁thought ▁provoking . ▁xxbos ▁i ▁have ▁been ▁hooked ▁on ▁\" gg \" ▁since ▁midway ▁through ▁2001 - 2002 ▁(2 nd ▁season ), ▁when ▁i ▁tuned ▁in ▁to ▁see ▁\" small ville \" ▁10 ▁minutes ▁early . ▁xxmaj ▁thanks ▁to</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>▁came ▁to ▁the ▁cinemas , ▁and ▁brought ▁my ▁entire ▁family ▁along . ▁i ▁had ▁already ▁seen ▁xxmaj ▁jim ▁xxmaj ▁carrey ▁in ▁the ▁xxmaj ▁mask , ▁and ▁xxmaj ▁ace ▁xxmaj ▁ventura , ▁and ▁loved ▁him ▁in ▁those ▁films . ▁xxmaj ▁the ▁review ▁of ▁the ▁film ▁was ▁quite ▁good , ▁so ▁i ▁looked ▁forward ▁to ▁this . ▁xxmaj ▁my ▁father ▁wondered ▁if ▁it ▁really ▁was ▁a ▁movie ▁in ▁his ▁taste ▁... ▁and ▁then ▁the ▁movie ▁started . ▁i ▁have ▁never ▁in ▁my</td>\n",
       "      <td>▁to ▁the ▁cinemas , ▁and ▁brought ▁my ▁entire ▁family ▁along . ▁i ▁had ▁already ▁seen ▁xxmaj ▁jim ▁xxmaj ▁carrey ▁in ▁the ▁xxmaj ▁mask , ▁and ▁xxmaj ▁ace ▁xxmaj ▁ventura , ▁and ▁loved ▁him ▁in ▁those ▁films . ▁xxmaj ▁the ▁review ▁of ▁the ▁film ▁was ▁quite ▁good , ▁so ▁i ▁looked ▁forward ▁to ▁this . ▁xxmaj ▁my ▁father ▁wondered ▁if ▁it ▁really ▁was ▁a ▁movie ▁in ▁his ▁taste ▁... ▁and ▁then ▁the ▁movie ▁started . ▁i ▁have ▁never ▁in ▁my ▁movie</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>▁or ▁xxmaj ▁ladd ▁and ▁xxmaj ▁lake ) to ▁soften ▁the ▁violent ▁elements . ▁\" kiss ▁xxmaj ▁tomorrow ▁xxmaj ▁goodbye \" ▁( a ▁gem ▁of ▁a ▁noir ▁title ▁if ▁there ▁ever ▁was ▁one , ▁be speaking ▁a ▁bleak , ▁fatalistic ▁vision ) ▁contains ▁no ▁romantic ▁subplot , ▁unless ▁you ▁count ▁the ▁xxmaj ▁cagney ▁character ' s ▁involvement ▁with ▁the ▁rich ▁woman , ▁which ▁is ▁more ▁about ▁greed ▁and ▁rebellion ▁than ▁love . ▁xxmaj ▁this ▁film ▁is ▁a ▁great ▁of ▁example ▁of</td>\n",
       "      <td>▁xxmaj ▁ladd ▁and ▁xxmaj ▁lake ) to ▁soften ▁the ▁violent ▁elements . ▁\" kiss ▁xxmaj ▁tomorrow ▁xxmaj ▁goodbye \" ▁( a ▁gem ▁of ▁a ▁noir ▁title ▁if ▁there ▁ever ▁was ▁one , ▁be speaking ▁a ▁bleak , ▁fatalistic ▁vision ) ▁contains ▁no ▁romantic ▁subplot , ▁unless ▁you ▁count ▁the ▁xxmaj ▁cagney ▁character ' s ▁involvement ▁with ▁the ▁rich ▁woman , ▁which ▁is ▁more ▁about ▁greed ▁and ▁rebellion ▁than ▁love . ▁xxmaj ▁this ▁film ▁is ▁a ▁great ▁of ▁example ▁of ▁pure</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>▁xxmaj ▁swann ▁( o ' toole ) ▁out ▁of ▁trouble ▁ . ▁xxmaj ▁by ▁no ▁means ▁an ▁easy ▁task . ▁xxmaj ▁the ▁things ▁they ▁do ▁together ▁are ▁quite ▁out ▁of ▁the ▁ordinary . ▁xxmaj ▁though ▁he ▁is ▁in ▁charge ▁of ▁keeping ▁xxmaj ▁swann ▁out ▁of ▁trouble , ▁he ▁really ▁gets ▁to ▁enjoy ▁his ▁job . ▁xxmaj ▁this ▁film ▁also ▁show ▁the ▁side ▁of ▁the ▁entertainment ▁history ▁we ▁don ' t ▁see , ▁xxmaj ▁swann ▁( o ' toole ▁ )</td>\n",
       "      <td>▁swann ▁( o ' toole ) ▁out ▁of ▁trouble ▁ . ▁xxmaj ▁by ▁no ▁means ▁an ▁easy ▁task . ▁xxmaj ▁the ▁things ▁they ▁do ▁together ▁are ▁quite ▁out ▁of ▁the ▁ordinary . ▁xxmaj ▁though ▁he ▁is ▁in ▁charge ▁of ▁keeping ▁xxmaj ▁swann ▁out ▁of ▁trouble , ▁he ▁really ▁gets ▁to ▁enjoy ▁his ▁job . ▁xxmaj ▁this ▁film ▁also ▁show ▁the ▁side ▁of ▁the ▁entertainment ▁history ▁we ▁don ' t ▁see , ▁xxmaj ▁swann ▁( o ' toole ▁ ) ▁confesses</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>, ▁it ▁just ▁keeps ▁plodding ▁on . ▁xxmaj ▁christopher ▁xxmaj ▁walken ▁has ▁a ▁part , ▁but ▁it ▁is ▁completely ▁senseless , ▁as ▁is ▁most ▁of ▁the ▁movie . ▁xxmaj ▁this ▁movie ▁had ▁potential , ▁but ▁it ▁looks ▁like ▁some ▁really ▁bad ▁made ▁for ▁xxup ▁tv ▁movie . ▁i ▁would ▁avoid ▁this ▁movie . ▁xxbos ▁\" men ace \" ▁is ▁not ▁funny . ▁xxmaj ▁it ▁tries ▁hard ▁- ▁too ▁hard . ▁but ▁rarely ▁brings ▁a ▁smile . ▁xxmaj ▁there ▁is ▁no</td>\n",
       "      <td>▁it ▁just ▁keeps ▁plodding ▁on . ▁xxmaj ▁christopher ▁xxmaj ▁walken ▁has ▁a ▁part , ▁but ▁it ▁is ▁completely ▁senseless , ▁as ▁is ▁most ▁of ▁the ▁movie . ▁xxmaj ▁this ▁movie ▁had ▁potential , ▁but ▁it ▁looks ▁like ▁some ▁really ▁bad ▁made ▁for ▁xxup ▁tv ▁movie . ▁i ▁would ▁avoid ▁this ▁movie . ▁xxbos ▁\" men ace \" ▁is ▁not ▁funny . ▁xxmaj ▁it ▁tries ▁hard ▁- ▁too ▁hard . ▁but ▁rarely ▁brings ▁a ▁smile . ▁xxmaj ▁there ▁is ▁no ▁acting</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>▁pink ▁xxmaj ▁panther . ▁xxmaj ▁this ▁was ▁an ▁xxup ▁rko ▁movie ▁but ▁it ▁did ▁not ▁have ▁the ▁nice ▁airplane ▁logo ▁that ▁xxup ▁rko ▁used ▁to ▁use . ▁i ▁liked ▁xxmaj ▁victor ▁xxmaj ▁mature ▁in ▁xxmaj ▁one ▁xxmaj ▁million , ▁xxup ▁b . c . , ▁and ▁xxmaj ▁sam p son ▁and ▁xxmaj ▁delilah ▁and ▁especially ▁in ▁xxmaj ▁violent ▁xxmaj ▁saturday . ▁xxmaj ▁see ▁if ▁you ▁can ▁find ▁that ▁one . ▁xxmaj ▁he ▁was ▁wonderful ▁in ▁the ▁comedy ▁with ▁xxmaj ▁peter</td>\n",
       "      <td>▁xxmaj ▁panther . ▁xxmaj ▁this ▁was ▁an ▁xxup ▁rko ▁movie ▁but ▁it ▁did ▁not ▁have ▁the ▁nice ▁airplane ▁logo ▁that ▁xxup ▁rko ▁used ▁to ▁use . ▁i ▁liked ▁xxmaj ▁victor ▁xxmaj ▁mature ▁in ▁xxmaj ▁one ▁xxmaj ▁million , ▁xxup ▁b . c . , ▁and ▁xxmaj ▁sam p son ▁and ▁xxmaj ▁delilah ▁and ▁especially ▁in ▁xxmaj ▁violent ▁xxmaj ▁saturday . ▁xxmaj ▁see ▁if ▁you ▁can ▁find ▁that ▁one . ▁xxmaj ▁he ▁was ▁wonderful ▁in ▁the ▁comedy ▁with ▁xxmaj ▁peter ▁xxmaj</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dbunch_lm.show_batch()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we have a convenience method to directly grab a `Learner` from it, using the `AWD_LSTM` architecture."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "opt_func = partial(Adam, wd=0.1)\n",
    "learn = language_model_learner(dbunch_lm, AWD_LSTM, opt_func=opt_func, metrics=[accuracy, Perplexity()], path=path)\n",
    "learn = learn.to_fp16()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>perplexity</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.426135</td>\n",
       "      <td>3.984901</td>\n",
       "      <td>0.292371</td>\n",
       "      <td>53.779987</td>\n",
       "      <td>07:00</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(1, 2e-2, moms=(0.8,0.7,0.8))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save('stage1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.load('stage1');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>perplexity</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>4.163227</td>\n",
       "      <td>3.870354</td>\n",
       "      <td>0.306840</td>\n",
       "      <td>47.959347</td>\n",
       "      <td>07:24</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>4.055693</td>\n",
       "      <td>3.790802</td>\n",
       "      <td>0.316436</td>\n",
       "      <td>44.291908</td>\n",
       "      <td>07:41</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>2</td>\n",
       "      <td>3.979279</td>\n",
       "      <td>3.729021</td>\n",
       "      <td>0.323357</td>\n",
       "      <td>41.638317</td>\n",
       "      <td>07:22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>3</td>\n",
       "      <td>3.919654</td>\n",
       "      <td>3.688891</td>\n",
       "      <td>0.327770</td>\n",
       "      <td>40.000469</td>\n",
       "      <td>07:22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>4</td>\n",
       "      <td>3.889432</td>\n",
       "      <td>3.660633</td>\n",
       "      <td>0.330762</td>\n",
       "      <td>38.885933</td>\n",
       "      <td>07:22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>5</td>\n",
       "      <td>3.842923</td>\n",
       "      <td>3.637397</td>\n",
       "      <td>0.333315</td>\n",
       "      <td>37.992798</td>\n",
       "      <td>07:26</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>6</td>\n",
       "      <td>3.813823</td>\n",
       "      <td>3.619074</td>\n",
       "      <td>0.335308</td>\n",
       "      <td>37.303013</td>\n",
       "      <td>07:25</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>7</td>\n",
       "      <td>3.793213</td>\n",
       "      <td>3.608010</td>\n",
       "      <td>0.336566</td>\n",
       "      <td>36.892574</td>\n",
       "      <td>07:20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>8</td>\n",
       "      <td>3.766456</td>\n",
       "      <td>3.602140</td>\n",
       "      <td>0.337257</td>\n",
       "      <td>36.676647</td>\n",
       "      <td>07:22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>9</td>\n",
       "      <td>3.759768</td>\n",
       "      <td>3.600955</td>\n",
       "      <td>0.337450</td>\n",
       "      <td>36.633202</td>\n",
       "      <td>07:23</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.unfreeze()\n",
    "learn.fit_one_cycle(10, 2e-3, moms=(0.8,0.7,0.8))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have fine-tuned the pretrained language model to this corpus, we save the encoder since we will use it for the classifier."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn.save_encoder('finetuned1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use it to train a classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "texts = get_files(path, extensions=['.txt'], folders=['train', 'test'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "splits = GrandparentSplitter(valid_name='test')(texts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For classification, we need to use two set of transforms: one to numericalize the texts and the other to encode the labels as categories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_tfms = [Tokenizer.from_folder(path), Numericalize(vocab=dbunch_lm.vocab)]\n",
    "dsets = Datasets(texts, [x_tfms, [parent_label, Categorize()]], splits=splits, dl_type=SortedDL)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "bs = 64"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "dls = dsets.dataloaders(before_batch=pad_input_chunk, bs=bs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>category</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>xxbos * * attention xxmaj spoilers * * \\n\\n xxmaj first of all , let me say that xxmaj rob xxmaj roy is one of the best films of the 90 's . xxmaj it was an amazing achievement for all those involved , especially the acting of xxmaj liam xxmaj neeson , xxmaj jessica xxmaj lange , xxmaj john xxmaj hurt , xxmaj brian xxmaj cox , and xxmaj tim xxmaj roth . xxmaj michael xxmaj canton xxmaj jones painted a wonderful portrait of the honor and dishonor that men can represent in themselves . xxmaj but alas … \\n\\n it constantly , and unfairly gets compared to \" braveheart \" . xxmaj these are two entirely different films , probably only similar in the fact that they are both about xxmaj scots in historical xxmaj scotland . xxmaj yet , this comparison frequently bothers me because it seems</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>xxbos xxmaj by now you 've probably heard a bit about the new xxmaj disney dub of xxmaj miyazaki 's classic film , xxmaj laputa : xxmaj castle xxmaj in xxmaj the xxmaj sky . xxmaj during late summer of 1998 , xxmaj disney released \" kiki 's xxmaj delivery xxmaj service \" on video which included a preview of the xxmaj laputa dub saying it was due out in \" 1 xxrep 3 9 \" . xxmaj it 's obviously way past that year now , but the dub has been finally completed . xxmaj and it 's not \" laputa : xxmaj castle xxmaj in xxmaj the xxmaj sky \" , just \" castle xxmaj in xxmaj the xxmaj sky \" for the dub , since xxmaj laputa is not such a nice word in xxmaj spanish ( even though they use the word xxmaj laputa many times</td>\n",
       "      <td>pos</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "dls.show_batch(max_n=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we once again have a convenience function to create a classifier from this `DataLoaders` with the `AWD_LSTM` architecture."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "opt_func = partial(Adam, wd=0.1)\n",
    "learn = text_classifier_learner(dls, AWD_LSTM, metrics=[accuracy], path=path, drop_mult=0.5, opt_func=opt_func)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We load our pretrained encoder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "learn = learn.load_encoder('finetuned1')\n",
    "learn = learn.to_fp16(clip=0.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Then we can train with gradual unfreezing and differential learning rates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lr = 1e-1 * bs/128"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.328318</td>\n",
       "      <td>0.200650</td>\n",
       "      <td>0.922120</td>\n",
       "      <td>01:08</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.fit_one_cycle(1, lr, moms=(0.8,0.7,0.8), wd=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.208120</td>\n",
       "      <td>0.166004</td>\n",
       "      <td>0.937440</td>\n",
       "      <td>01:15</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.freeze_to(-2)\n",
    "lr /= 2\n",
    "learn.fit_one_cycle(1, slice(lr/(2.6**4),lr), moms=(0.8,0.7,0.8), wd=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.162498</td>\n",
       "      <td>0.154959</td>\n",
       "      <td>0.942400</td>\n",
       "      <td>01:35</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.freeze_to(-3)\n",
    "lr /= 2\n",
    "learn.fit_one_cycle(1, slice(lr/(2.6**4),lr), moms=(0.8,0.7,0.8), wd=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: left;\">\n",
       "      <th>epoch</th>\n",
       "      <th>train_loss</th>\n",
       "      <th>valid_loss</th>\n",
       "      <th>accuracy</th>\n",
       "      <th>time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <td>0</td>\n",
       "      <td>0.133800</td>\n",
       "      <td>0.163456</td>\n",
       "      <td>0.940560</td>\n",
       "      <td>01:34</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <td>1</td>\n",
       "      <td>0.095326</td>\n",
       "      <td>0.154301</td>\n",
       "      <td>0.945120</td>\n",
       "      <td>01:34</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "learn.unfreeze()\n",
    "lr /= 5\n",
    "learn.fit_one_cycle(2, slice(lr/(2.6**4),lr), moms=(0.8,0.7,0.8), wd=0.1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "jupytext": {
   "split_at_heading": true
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
